Personal augmented reality

ABSTRACT

Image data comprising an image of an environment is received from a mobile video device located at the environment. An image of a marker is extracted from the received image for use in identifying the location of the mobile video device. A location associated with the marker is extracted from a database and the extracted location is sent as the location of the mobile video device. Also received is a request for information about an item in the environment that is of interest to a user of the mobile video device. A location in the environment associated with the item of interest is identified, and the information indicating the location of the item is transmitted to the mobile device for display within an image of the environment on the mobile video device. The item of interest may be displayed as an augmented reality within an image of the environment.

CROSS REFERENCE TO RELATED APPLICATION

This application is a continuation of and claims the benefit of priorityto U.S. patent application Ser. No. 14/963,706, filed Dec. 9, 2015,which is a continuation of and claims the benefit of priority to U.S.patent application Ser. No. 13/340,141, filed on Dec. 29, 2011, now U.S.Pat. No. 9,240,059, the benefit of priority of each of which is claimedhereby, and each of which are incorporated by reference herein in theirentirety.

TECHNICAL FIELD

This application relates to a method and system for providing augmentedreality for the environment in which a user is watching an event live ina stadium.

BACKGROUND

An individual viewing a real-time event in person may desire informationrelated to the real-time event, the people engaged in the real-timeevent, the locale of the real-time event, and even other viewers at thelocale of the real-time event. Further, the viewing individual maydesire that information in such as way as to augment his or her viewingof the event. Conventionally, the only such information available to theviewer is merely the fraction of the desired information, for examplethat information which entities controlling the real-time event mayelect to provide at a scoreboard or other display at the real-timeevent. Even this information does not actually augment the real timeevent other than by being viewable in a general, and not personal (suchas personal choice), way. Consequently, the individual viewer has no, oronly a very limited, way of obtaining the desired information presentedin a way that is most beneficial to that individual. In addition, salesorganizations desire to target advertisements to the viewing individualat the locale, based on characteristics or location of the individual.Often, such organizations have only very limited ways of targeting suchadvertisements.

BRIEF DESCRIPTION OF THE DRAWINGS

The present disclosure is illustrated by way of example, and not by wayof limitation, in the figures of the accompanying drawings in which:

FIG. 1 is a network diagram depicting a network system, according to oneembodiment, having a client-server architecture configured forexchanging data over a network;

FIG. 2 is a block diagram illustrating an example embodiment of alocation-based incentive application;

FIG. 3 is a block diagram illustrating an example embodiment of alocation identification module;

FIG. 4 is a block diagram illustrating an example embodiment of an itemidentification module;

FIG. 5 is a block diagram illustrating an example embodiment of anincentive module;

FIG. 6 is a flow chart illustrating a method for augmenting personal,real-time, viewing of an event by a user at the location of the event;

FIG. 7A is an illustration of a ballgame at a ballpark;

FIG. 7B is an illustration of the ball game of FIG. 7A with statisticsof the player on second base, as requested by the user;

FIG. 8 is an illustration of seating and other locations in a ballpark;

FIG. 9A is an illustration of one embodiment of personal augmentationfor a user;

FIG. 9B is an illustration of augmenting the personal view of a userwith the personal augmentation of FIG. 9A; and

FIG. 10 shows a diagrammatic representation of machine in the exampleform of a computer system within which a set of instructions may beexecuted to cause the machine to perform any one or more of themethodologies discussed herein.

DETAILED DESCRIPTION

Although the present disclosure has been described with reference tospecific example embodiments, it will be evident that variousmodifications and changes may be made to these embodiments withoutdeparting from the broader spirit and scope of the disclosure.Accordingly, the specification and drawings are to be regarded in anillustrative rather than a restrictive sense.

Example embodiments described herein provide systems and methods foraugmenting one's experience at a real time in-person attendance of anevent such as, in one embodiment, attending a baseball game at aballpark. This may be accomplished by concurrently viewing the event atthe ballpark using a camera such as on an iPhone, or a camera embodiedin a mobile device such as an iPad. Such devices may herein be termed a“client device” or a “user device” or an iPhone or an iPad withoutlimiting the type of device used. In example embodiments, environmentimage data containing an image of an environment is received by a systemfrom a client device operated by a user, in one embodiment, viewing theball game at the ballpark.

The user may wish to use the user device to watch the game withstatistics the user is most interested in, such as statistics for oneteam, for one particular player, for one position, and the like. Theuser may also desire to have an embodiment which may be used todetermine specific attributes of the ball park, such as type andlocation of a restaurant, handicapped bathrooms, an exit closest towhere the user parked, and the like. The user may send the system arequest for the desired information to augment the reality of the ballgame being watched at the ballpark.

Likewise, marketing organizations, whether online ecommerceorganizations or conventional “brick and mortar” organizations, maydesire an embodiment that allows them to target ads to the user. Theseads may be for items that are based on the user's purchase history,interests, or on the location of an organization closest to the ballparkthat sells such items. Given different histories and interests ofpeople, two people could be watching the same game at the same ballparkhut have different ads targeted to them. Using appropriate imagescaling, a selection of statistic or ad to be targeted, and itsplacement into an indicated location of the environment as anaugmentation, may be implemented. As used herein, the statistic ortargeted ad may be referred to herein as an “item” for ease ofreference, and without losing generality. An image of the selected itemis scaled to a scale that is based on dimensions determined from theenvironment image data for the environment. The dimensions may bedetermined based on a calculated distance to a focal point of theindicated location in the environment and/or on a marker located in theimage of the environment. The scaled item image is augmented into theimage of the environment at the indicated location to generate anaugmented reality image. In some embodiments, the scaled item may beoriented to match an orientation of the indicated location in theenvironment. Technology useful in implementing augmented reality may beseen in U.S. patent application Ser. No. 13/283,416 filed Oct. 27, 2011and entitled “SYSTEM AND METHOD FOR VISUALIZATION OF ITEMS IN ANENVIRONMENT USING AUGMENTED REALITY,” particularly FIGS. 1-5 thereof,and the text associated therewith. The foregoing patent application isowned by the owner of the present patent and is hereby incorporated byreference herein in its entirety.

By using embodiments of the present disclosure, a user may search for anitem and augment an image of an environment with an image of the itemthat is a personal choice of the user. Because the user may create andview an augmented reality image of the environment including theselected item, the user easily views the selected item in theenvironment. Therefore, one or more of the methodologies discussedherein may obviate a need for time consuming data processing by the userin order to obtain requested statistics and the like. This may have thetechnical effect of reducing computing resources used by one or moredevices within the system. Examples of such computing resources include,without limitation, processor cycles, network traffic, memory usage,storage space, and power consumption.

Embodiments may be implemented using wireless or other suitabletechnology. As discussed, the user may be watching a live baseball gamein a ballpark. The system may have stored in its database relevantinformation about ballparks of interest, the information includinglocation of various points within each ballpark, perhaps by ballparkcoordinates, or by other appropriate locating markers such as ballparksection numbers and seat numbers. The system may also have stored in itsdatabase the players of relevant teams (e.g., teams in the league inwhich the current team plays), by number, position, or otheridentification, and their statistics. In the discussion that follows,queries or requests that may be made to the system by the user may beimplemented as selection of predetermined queries available on a userinterface (“UI”) on a user device such as a UI of an iPad or an iPhoneon which the user is watching the game live in the ballpark. Forexample, the user may select predetermined requests on the UI. This mayalso be accomplished, for example, by the user employing a UI on theuser device to send a non-predetermined request (e.g., a request whichis of the user's personal choice) to the system for statistics for theplayer being viewed, whose identity may also be sent to the system bythe user by way of the UI, or may be otherwise recognized by the systemby the technologies discussed more fully below. Similarly, the systemdatabase may have stored the team pitching rotation for starters andrelievers for this and, perhaps, all relevant teams, and any playersthat are currently on the injured reserve list for each team. Thedatabase may also store the current league standing and the possibleplayoff scenarios based on current won/lost for each team, and thepossibilities of wins and losses for teams leading up to the playoffs.

In one embodiment, the user may use the camera of a user device such asan iPhone or an iPad to provide context, such as the baseball game theuser is watching in a ballpark, as mentioned above, along with requesteditems of information. In response, the system and the display of theuser device may show items popping up to augment the reality of the gamethat is being watched. While the user is looking at the field using aniPad, he or she may aim the camera of the iPad for a close-up of aparticular player. The player's image may be rendered on the display ofthe iPad during the real time baseball game, and player statistics forthat player may also be rendered on the display by the system,responsive to the user's request for the statistics. This may beaccomplished, for example, by the user employing a UI on the user deviceto send a request to the system for statistics for the player beingviewed, whose identity may be sent to the system by the user by way ofthe UI, or otherwise recognized by the system as more fully discussedbelow.

In another embodiment, the system may include a video recognition systemthat recognizes images of the players, for example, by the player's teamuniform number, to enable the system to correlate team, uniform number,and player in order to transmit the correct statistics for viewing if sorequested. The system may, if desired, point to the particular player byarrow or other indicator when augmenting the reality of in-personwatching of the game live, by presenting the player's statistics on theuser device.

In another embodiment, the camera of the iPad may point from player toplayer and the system may recognize the players, by number recognition,or by other technology recognition such as facial recognitionalgorithms, or by RFID identification from an RFID chip embedded in theplayer's uniform. The user's favorite players may be found by matchingan earlier prepared list of the user's favorite players located in anappropriate database, including the players' numbers or otheridentification, with the player identity that is so-recognizable by thesystem as the camera pans across the field. The recognition may be madeby using the technology discussed above, the numbers being compared tothose of the user's favorite players by database search as the userdevice scans a player. The system may then transmit the requested playerstatistics for rendering on the iPad display along with, for example,arrows pointing to the various favorite players on the screen of theuser device as the camera pans across the field of play. Similarly, theball itself may contain an RFID chip, and the above technology used toaugment the reality of the ballgame on the iPad by using the detectedpath of the RFID chip within the ball to illustrate the trajectory ofthe ball.

Based on the above implementations, the user may request the statisticsof a particular player; the statistics of his or her favorite player;how a player compares in various statistic areas with other players atthe same position; how the particular player compares in battingaverage; or how the player compares in other statistics categories withother players in the league.

Another embodiment may show the user, for example by arrows, wherefriends of the user are seated in the stands of the ballpark. This maybe implemented, as one example, by the user sending the system a queryfor the seat location of a friend. Responsive to this query, the systemmay read the friend's phone number from the user's cell phone contactlist and, in conjunction with a global positioning service (“GPS”)satellite system, determine the seat location, generally orspecifically, of the cell phone that has the friend's phone number. Thesystem may then, from the above stored ballpark coordinates, respond tothe user with information as to the friend's seat location, including,if desired, placing an arrow on the display of the user device pointingto the location of the friend's seat, augmenting the viewer's personalreality of viewing the ballpark live. In an alternate embodiment,instead of GPS technology, the system may have stored in its databasethe names of people who purchased the various seats in the ballpark, andmay then, upon request for the seating of a particular person, searchfor the name of that person to find the person's seat location. Again,if desired, an arrow pointing to the seat location may be placed uponthe image of the user device in order to augment the reality of the liveviewing of the game. In yet another embodiment, the friend may haveposted on a social network such as Facebook the section, row, and seatnumber where that friend will be sitting. The user may then read thisseating off the friend's Facebook wall and send that information to thesystem, for use by the system in augmenting the user device by, as oneexample, pointing on the device's display to the point in the ballparkwhere that seating is located. In still another embodiment, directionsmay be provided based on the user's preferences sent to the system withor as part of the request. For example, if the user requires elevatorsthe system will point the user in the direction that includes elevators,if the user needs a ramp, the system points to the direction includingramps. Otherwise the system may be free to point out directions usingstairs.

In another embodiment, the iPhone or iPad may function as a “set ofeyeglasses,” providing the user with information relevant to the user'spersonal choices as the user moves the user device from place to placewithin the ballpark. For example, as the user walks out of the user'sseating section towards the ballpark concourse the user device may,responsive to user query, show the location of ballpark locations thatthe user may be interested in, for example the closest bathroom, or abathroom that is outfitted for the handicapped. This could be done bythe camera of the user device transmitting to the system images ofvarious locations of the ballpark, such as a marker showing a ballparksection number. As mentioned previously, the system may have previouslystored in a database an image of the ballpark, with coordinates forvarious areas of the ballpark, perhaps keyed to various ballparkmarkers. The database may be searched using the marker transmitted fromthe camera, and the desired location may be rendered on the display ofthe user device along with the pointing information discussed above.

Other embodiments may include the user requesting the system to displaythe shortest path out of the stadium to where the user parked his or hercar. The system may receive a feed of road traffic, and the user couldrequest the parking lot exit from which to leave in order to encounterthe least traffic, the fastest way home, or the route that has the leasttraffic, or the route that passes a particular restaurant, or thatpasses several restaurants on the way home, and the system could respondwith that information for display on the user device.

In any of the above embodiments, as desired, the system may also accessthe user's purchasing history it for example, the system is, or isassociated with, an ecommerce system or other type of marketing system.The system may then along with, or separate from, the above augmentedreality information, transmit advertisements to the user based on thatpurchasing history. In addition, the advertisements may be based on thelocation of the ballpark and indicate the location of a store near theballpark that sells the subject matter of the advertisement. The systemmay, when sending statistics that were requested for a player, alsoinclude an advertisement showing where the jersey or other souvenirs ofthe player can be purchased, or where team memorabilia, such aschampionship pennant replicas may be purchased. Since the purchasinghistory is personal to the user, different users using the above methodswould view the same game in the same ballpark, but would receivedifferent responses to the same query from the system.

FIG. 1 is a network diagram depicting a network system 100, according toone embodiment, having a client-server architecture configured forexchanging data over a network. For example, the network system 100 maybe a publication/publisher system 102 where clients may communicate andexchange data within the network system 100. The data may pertain tovarious functions (e.g., online item purchases) and aspects (e.g.,managing content and user reputation values) associated with the networksystem 100 and its users. Although illustrated herein as a client-serverarchitecture as an example, other embodiments may include other networkarchitectures, such as a peer-to-peer or distributed networkenvironment.

A data exchange platform, in an example form of a network-basedpublisher 102, may provide server-side functionality, via a network 104(e.g., the Internet) to one or more clients. The one or more clientsinclude users that utilize the network system 100 and more specifically,the network-based publisher 102, to exchange data over the network 114.These transactions may include transmitting, receiving (communicating)and processing data to, from, and regarding content and users of thenetwork system 100. The data may include, but are not limited to,content and user data such as feedback data; user reputation values;user profiles; user attributes; product and service reviews andinformation, such as pricing and descriptive information; product,service, manufacture, and vendor recommendations and identifiers;product and service listings associated with buyers and sellers; auctionbids; and transaction data, among other things.

In various embodiments, the data exchanges within the network system 100may be dependent upon user-selected functions available through one ormore client or UIs. The UIs may be associated with a client machine,such as a client machine 106 using a web client 110. The web client 110may be in communication with the network-based publisher 102 via a webserver 120. The UIs may also be associated with a client machine 108using a programmatic client 112, such as a client application, or amobile device 132 hosting a third party application 116. It may beappreciated in various embodiments the client machine 106, 108, or thirdparty server 114 may be associated with a buyer, a seller, a third partyelectronic commerce platform, a payment service provider, or a shippingservice provider, each in communication with the network-based publisher102 and optionally each other. The buyers and sellers may be any one ofindividuals, merchants, or service providers, among other things.

A mobile device 132, such as an iPad or an iPhone, as non-limitingexamples, may also be in communication with the network-based publisher102 via a web server 120. The mobile device 132 may include a portableelectronic device providing at least some of the functionalities of theclient machines 106 and 108. The mobile device 132 may include a thirdparty application 116 (or a web client) configured communicate withapplication server 122.

Turning specifically to the network-based publisher 102, an applicationprogram interface (API) server 118 and a web server 120 are coupled to,and provide program and web interfaces respectively to, one or moreapplication servers 122. The application servers 122 may host one ormore publication application(s) 124 and a location-based incentiveapplication 130. The application servers 122 are, in turn, shown to becoupled to one or more database server(s) 126 that facilitate access toone or more database(s) 128.

In one embodiment, the web server 120 and the API server 118 communicateand receive data pertaining to statistics, and feedback, among otherthings, via various user input tools. For example, the web server 120may send and receive data to and from a toolbar or webpage on a browserapplication (e.g., web client 110) operating on a client machine (e.g.,client machine 106). The API server 118 may send and receive data to andfrom an application (e.g., client application 112 or third partyapplication 116) running on other client machines (e.g., client machine108 or mobile device 132 which, as previously indicated, may he aniPhone or an iPad).

A publication application(s) 124 may provide a number of publisherfunctions and services (e.g., listing, payment, etc.) to users thataccess the network-based publisher 102. For example, the publicationapplication(s) 124 may provide a number of services and functions tousers for listing goods and/or services for sale, facilitatingtransactions, and reviewing and providing feedback about transactionsand associated users. Other publication applications (not shown) withinor external to the network based publisher 102 may provide statisticsfor players, names of parties associated with ballpark seating, trafficinformation, map information, store locations, and the like.

The third party application 116 may execute on a third party server (notshown) and may have programmatic access to the network-based publisher102 via the programmatic interface provided by the API server 118. Forexample, the third party application 116 may use information retrievedfrom the network-based publisher 102 to support one or more features orfunctions on a website hosted by the third party. The third partywebsite may, for example, provide one or more listing, feedback,publisher or payment functions that are supported by the relevantapplications of the network-based publisher 102.

The network-based publisher 102 may provide a multitude of feedback,reputation, aggregation, and listing and price-setting mechanismswhereby a user may be a seller or buyer who lists or buys goods and/orservices (e.g., for sale) published on the network-based publisher 102.

The application server 122 also includes a location-based application130. The location-based application 130 communicates advertisements,some of which may offer incentives, to the mobile device 132 based onthe mobile device 132 location and the purchase history or otherindicated preferences of the user of the mobile device 132 as furtherdescribed below.

FIG. 2 is a block diagram illustrating an example embodiment of alocation-based incentive application 130, which is provided as part ofthe network-based publisher 102. The location-based incentiveapplication 130 has a location identification module 202, an itemidentification module 204, and an incentive module 206. The locationidentification module 202 determines a geographic location of the mobiledevice 132. The item identification module 204 identifies an item, suchas statistics for a player, or for a plurality of players, specified bythe user at the geographic location of the mobile device 132. Theincentive module 206 communicates an incentive from one or more localmerchants based on the identified item and the geographic location ofthe mobile device 132. These modules may be implemented in hardware,firmware, or any combination thereof. As outlined above, database 128may contain purchase history of the user of the mobile device, as wellas the aforementioned location.

In one embodiment, the location-based incentive application 130 receivesa communication from the mobile device 132. For example, thecommunication may include a specification of an item and a location ofthe mobile device 132. Based on the specified item and the location ofthe mobile device 132, the incentive module 206 queries to retrieveresults from the database server 126 and database 128 to determine andcommunicate incentives from local merchants to the mobile device 132.

FIG. 3 is a block diagram illustrating an example embodiment of thelocation identification module 202. The location of the mobile device132 may be determined in many ways. For example, the mobile device 132may be equipped with a GPS system that would allow the device tocommunicate the coordinates or location of the mobile device 132 to aGPS/triangulation module 302 of the location identification module 202.In another example, the location of the mobile device 132 may hedetermined by triangulation using wireless communication towers and/orwireless nodes e.g. wi-fi hotspots within wireless signal reach of themobile device 132. Based on the geographic coordinates, theGPS/triangulation module 302 of the location identification module 202may determine the geographic location of the mobile device 132 afterconsulting a mapping database (not shown). Furthermore, the generallocation of the mobile device 132 may be located when the user of themobile device 132 logs onto a local internet connection, for example, atthe above ballpark, at a hotel, at a coffee shop, or at any otherorganization based on location. The Internet Protocol address of thenetwork connection at the ballpark may be uniquely identified by thelocation of the ballpark.

The location identification module 202 may also include a location inputmodule 306 configured to determine a geographic location of the mobiledevice 132 by requesting the user to input an address, city, zip code orother location information on his/her mobile device 132. In oneembodiment, the user may select a location from a list of locations or amap on the mobile device 132. For example, a user on the mobile device132 inputs the location of the mobile device 132 via an application or aweb browser on the mobile device 132. In another embodiment, thelocation input module 306 derives the geographic location of the user bycommunicating with third party application using respective APIs(Application Programming Interface).

The location identification module 202 may also include alocation-dependent search term module 304. The location of the mobiledevice 132 may be inferred when the user of the mobile device 132requests a search on the mobile device 132 using location-dependentsearch terms. For example, a user inputs a request, sometimes referredherein as a “search query,” on his/her mobile device for “Best JapaneseRestaurant in my locale,” “exit nearest my seat,” “exit nearest where mycar is parked,” and the like. The location-dependent search term module304 queries and retrieves results from a database (not shown) that maydetermine the geographic location of the ballpark, the best Japaneserestaurant near the user's location, perhaps based on opinions submittedby restaurant customers. Further, the user may request, or the user'shistory (such as on a social network) may show, a preference for acertain type of Japanese food. The system may then send directions tothe closest Japanese restaurant that serves that preferred Japanesefood, and not merely the closest Japanese restaurant. Thelocation-dependent search term module 304 may have earlier received fromthe user the user's seating at the ballpark to use as needed for theabove requests, or may have obtained the user's seating by having accessto seats purchased by person, which information may also be used fordetermining the information to be supplied by the system responsive tothe request.

The location identification module 202 may also include a tag module 308configured to determine the geographic location of the mobile device 132based on a tag associated with a unique geographic location. The tag mayinclude, for example, a barcode tag (e.g. linear barcode or twodimensional bar code) or a Radio Frequency Identification (RFID) tagthat is associated with a unique geographic location. For example, theuser of the mobile device 132 may use his/her mobile device 132 to scanthe tag placed at a landmark or store. The tag may be uniquelyassociated with the geographic location of the landmark or store. Suchrelationship may be stored in a database. The tag module 308 may thendetermine the geographic location of the mobile device 132 based on thetag after consulting the database.

FIG. 4 is a block diagram illustrating an example embodiment of an itemidentification module 204. The item specified by the user of the mobiledevice 132 may be determined in many ways using any of the followingexamples of modules: a text identification module 402, an audioidentification module 404, a machine-readable symbol module 406, animage identification module 408, and a video identification module 410.The text identification module 402 may identity an item, such asstatistics of a given player, or statics of players at a given position,specified by the user at the mobile device 132, in one embodiment usinga text input from the user at the mobile device 132. For example, theuser may enter a request for a particular player or for a particularposition the user wishes. The text identification module 402 may furtheridentify the item by comparing the request for the statistics with adatabase containing the statistics by player. For example, the user mayspecify “statistics for David Ortiz” as a category for searching. Thetext identification module 402 may then identify statistics thatcorrespond to the text that was input by the user. In this case, thetext identification module 402 identifies statistics that match theplayer input by the user (e.g. David Ortiz).

The audio identification module 404 may identify an item or a categoryof the item as specified by the user at the mobile device using an audioinput from the user at the mobile device. For example, the user mayspeak “statistics” and “David Ortiz” at the location of the mobiledevice. The audio identification module 404 may include a speechrecognition system (not shown) that enables the spoken words of the userto be transcribed into text.

The audio identification module 404 may be used to identify thespecified statistics by comparing the name of the player transcribedfrom the audio with a database containing player statistics.

The machine-readable symbol module 406 may identify an item by havingthe user scan an RFID chip or any other machine-readable symbol such asa player's uniform number, at a distance, with his/her mobile device 132as a machine-readable symbol reader. For example, the mobile device 132may include an optical device (e.g. a lens) configured to capture animage of a player. The mobile device 132 may then upload the capturedimage to the machine-readable symbol module 406. The machine-readablesymbol module 406 processes the captured image by querying andretrieving results from a database of machine-readable images to matchthe captured image of the player symbol with corresponding statistics.The machine-readable symbol module 406 may then identify the statisticsspecified b the user at the mobile device for rendering at the userdevice to augment the viewing or watching of the ball game in real timeat the ballpark.

The image identification module 408 may identify an item by having theuser take a picture of the item with his/her mobile device 132. Mobiledevices commonly have an optical lens for capturing pictures. The mobiledevice 132 may then upload the picture, for example, of the player withcertain recognizable features to the image identification module 408.The image identification module 408 analyzes the picture using an imagerecognition algorithm such as a facial feature recognition algorithm forfacial recognition, or a numerical recognition algorithm jersey numberrecognition (neither algorithm shown) to match the uploaded picture witha corresponding image of an item. The image recognition algorithmconsults a database of images and corresponding statistics or otherinformation items to identify the uploaded picture.

The video identification module 410 may be configured to identify anitem by having the user take a video of the player with the user'smobile device. Mobile devices commonly have an optical lens to capturevideo. The mobile device 132 may then upload the video (or a portion ofthe video) to the video identification module 408. The videoidentification module 410 analyzes the frames of the video (for examplea jersey number or a player's facial image) using an image recognitionalgorithm (not shown) to match a frame of the video with a correspondingimage of the player. The image recognition algorithm may query andretrieve results from a database of images and corresponding items toidentify the uploaded video. For example, a user may take a video withhis/her mobile device of a player walking on the field. The videoidentification module 410 recognizes the player and identifies theplayer's statistics, among other identifying and descriptive informationabout the player. In any of the above embodiments, the system may alsoreturn an advertisement to the user based on the user's purchase historyif the system has access to that history. The system may also return anadvertisement to the user based on the user's interest, such as aninterest in a particular player. The system may also return anadvertisement based on the location of the user device.

FIG. 5 is a block diagram illustrating an example embodiment of theincentive module 206 that may used to execute the processes describedherein. The incentive module 206 may include a local merchant module502, an item module 504, an incentive match module 506, a preferencemodule 508, an incentive receiver module 510, and a communication module514.

The local merchant module 502 identifies at least one local merchanthaving at least one incentive based on the geographic location of themobile device 132 as determined by the location identification module202. A local merchant may be a merchant or retailer that is locatedwithin a predefined distance from the geographic location of the mobiledevice 132. In one embodiment, the local merchant module 502 identifiesat least one local merchant with at least one incentive based on asearch distance preference as specified in preference module 508.

It should be noted that the incentive of the local merchant may or maynot correspond to the item identified by the user. For example, a localmerchant may feature a special sale on shoes while the identified itemcorresponds to a digital camera. Once all local merchants havingincentives are identified based on the geographic location of the mobiledevice 132 (using a database of incentives), the incentive match module506 may filter all local merchants based on the identified item, such asa favorite player indicated by the user sending a request forinformation with respect to that player. In the previous example, thelocal merchant featuring a sale on sports memorabilia for the player mayhe filtered out from the search result. In addition, the filtering canbe on the basis of the user's personal shopping history, and/or alsopreferences. For example, if it is known, such as (in only one example)that the user's Facebook account indicates a number of “Likes” for hats,incentives for hats may be sent. If the information indicates a numberof “Dislikes” for shoes, incentives for shoes may not be sent.

The item category module 504 determines the item,in the current example,a player, specified by the user and identified by item identificationmodule 204. The item category module 504 determines that a category ofthe item related to the player specified by the user falls into thecategory of athletic memorabilia.

The incentive match module 506 determines whether the identified itemcategory corresponds to a category in at least one incentive of at leastone local merchant as determined by the local merchant module 502. Forexample, a user specifies David Ortiz with his/her mobile device. Theitem is identified as player David Ortiz. The item identification module204 generates an identity such as athletic memorabilia relating to DavidOrtiz. The local merchant module 502 identifies merchants withincentives local to the geographic location of the mobile device 132.The incentive match module 506 matches local merchants with incentives(sale or discount) on the specific memorabilia.

The communication module 514 communicates one or more incentives of theidentified item from at least one local merchant to the mobile device132. For example, a list of local merchants within a preset distanceradius (e.g. one mile) of the mobile device 132 is displayed. The listof local merchants may include a sale or discount on the item identifiedby the user of the mobile device 132. The list may also include a listof recommended merchants (having an incentive on the identified item)that are located beyond the preset distance radius.

In another embodiment, the communication module 514 communicates one ormore incentives of the identified category of the items from at leastone local merchant to the mobile device 132. For example, a list oflocal merchants within a preset distance radius (e.g. a block) of themobile device 132 is displayed. The list of local merchants may includea sale or discount on similar or related items to the identified itemspecified by the user of the mobile device 132. The list may alsoinclude a list of recommended merchants (having an incentive on similaritems to the identified item) that are located beyond the presetdistance radius.

FIG. 6 is a flowchart of a method 600, according to an embodiment, foraugmenting personal, real-time, in-person viewing of an event at alocation by a user. If an image of the environment, such as a ball parkand a ball game, and a request for an item for augmenting the reality ofthe user's real-time viewing, are received from the user device as at602, 604 (in any order) the system may detect the requested item andaccesses the item data as at 606 for use in augmenting the user'spersonal reality. This would include steps depending on the request. Forexample, the user may wish to use the user device to watch a baseballgame at a ballpark with augmentation comprising statistics the user ismost interested in, such as statistics for a particular player. In thiscase, the user device, for example an iPhone, might send the image ofFIG. 7A with the image of player number 34, David Ortiz, from the gamethe user is watching. The system may include a video identificationsystem 410 such as discussed above with respect to FIG. 4, thatrecognizes images of the players, for example, by the player's teamuniform number, so that the system may correlate team, uniform number,and player in order to transmit the correct statistics for viewing if sorequested. In this case the system may detect from the uniform number(and team, as appropriate) that the player is David Ortiz, and mayaccess the desired statistics for that player. The system may presentthe requested statistics as augmentation using the augmentationtechnology discussed above, with a result such as seen in FIG. 7B. Thesystem may, if desired, point to the particular player by arrow whenaugmenting the reality of watching the game live by presenting theplayer's statistics.

In another example of using the method of FIG. 6, the requested item at604 may be a request to show where a friend is seated. As discussedabove, the system ay use the GPS/triangulation technology 302 of FIG. 3to locate the requested seating. As an alternative, the system may useseating information provided from the user, or from the friend'sFacebook site to locate the seating. The system may then, using theaugmentation technology already discussed, or other such appropriatetechnology, augment the user's view of the real-time in-person viewingby placing an arrow, such as that seen in FIG. 9A, pointing to theseating on the display of the user device while the camera of the deviceis viewing the ballpark, as seen in FIG. 9B. The actual location of theseating may be determined by the system as discussed generally above, byusing ballpark coordinates, or by other appropriate locating markerssuch as ballpark section number and seat numbers, illustrated in FIG. 8,which may already be stored in the system's database. While the ballparkillustrated in FIG. 8 shows only the seating of the ballpark, well knowntechnology may be employed to superimpose a grid of coordinates on theimage of FIG. 8 and use the resultant map to locate the seating in orderto implement the augmentation under discussion.

Also as discussed above with respect to the location-based applications130, including incentive module 206, the system may use the user'slocation, to target advisements to the user. Technology for implementingadvertisement targeting is discussed in more detail in U.S. patentapplication Ser. No. 13/050,769 filed Mar. 17, 2011, and entitled“TARGETED INCENTIVE ACTIONS BASED ON LOCATION AND INTENT,” particularlyFIGS. 7A-7C and the text associated therewith. The foregoing applicationis owned by the assignee of the present patent and is herebyincorporated herein by reference in its entirety.

FIG. 8 shows a diagrammatic representation of machine in the exampleform of a computer system 800 within which a set of instructions may beexecuted causing the machine to perform any one or more of themethodologies discussed herein. In alterative embodiments, the machineoperates as a standalone device or may be connected (e.g., networked) toother machines. In a networked deployment, the machine may operate inthe capacity of a server or a client machine in server-client networkenvironment, or as a peer machine in a peer-to-peer (or distributed)network environment. The machine may be a personal computer (PC), atablet PC, a set-top box (STB), a Personal Digital Assistant (PDA), acellular telephone, a web appliance, a network router, switch or bridge,or any machine capable of executing a set of instructions (sequential orotherwise) that specify actions to be taken by that machine. Further,while only a single machine is illustrated, the term “machine” shallalso be taken to include any collection of machines that individually orjointly execute a set (or multiple sets) of instructions to perform anyone or more of the methodologies discussed herein.

The example computer system 800 includes a processor 802 (e.g., acentral processing unit (CPU), a graphics processing unit (GPU) orboth), a main memory 804 and a static memory 806, which communicate witheach other via a bus 808. The computer system 800 may further include avideo display unit 810 (e.g., a liquid crystal display (LCD) or acathode ray tube (CRT)). The computer system 800 also includes analphanumeric input device 812 (e.g., a keyboard), a user interface (UI)navigation device 814 (e.g., a mouse), a disk drive unit 816, a signalgeneration device 818 (e.g., a speaker) and a network interface device820.

The disk drive unit 816 includes a machine-readable medium 822 on whichis stored one or more sets of instructions and data structures e.g.,software 824) embodying or utilized by any one or more of themethodologies or functions described herein. The software 824 may alsoreside, completely or at least partially, within the main memory 804and/or within the processor 802 during execution thereof by the computersystem 800, the main memory 804 and the processor 802 also constitutingmachine-readable media.

The software 824 may further be transmitted or received over a network826 via the network interface device 820 utilizing any one of a numberof well-known transfer protocols (e.g., HTTP).

While the machine-readable medium 822 is shown in an example embodimentto be a single medium, the term “machine-readable medium” should betaken to include a single medium or multiple media (e.g., a centralizedor distributed database, and/or associated caches and servers) thatstore the one or more sets of instructions. The term “machine-readablemedium” shall also be taken to include any medium that is capable ofstoring, encoding or carrying a set of instructions for execution by themachine and that cause the machine to perform any one or more of themethodologies of the present disclosure, or that is capable of storing,encoding or carrying data structures utilized by or associated with sucha set of instructions. The term “machine-readable medium” shallaccordingly be taken to include, hut not he limited to, solid-statememories, optical media, and magnetic media.

The Abstract of the Disclosure is provided to comply with 37 C.F.R.§1.72(b), requiring an abstract that will allow the reader to quicklyascertain the nature of the technical disclosure. It is submitted withthe understanding that it will not be used to interpret or limit thescope or meaning of the claims. In addition, in the foregoing DetailedDescription, it can he seen that various features are grouped togetherin a single embodiment for the purpose of streamlining the disclosure.This method of disclosure is not to be interpreted as reflecting anintention that the claimed embodiments require more features than areexpressly recited in each claim. Rather, as the following claimsreflect, the disclosed subject matter lies in less than all features ofa single disclosed embodiment. Thus the following claims are herebyincorporated into the Detailed Description, with each claim standing onits own as a separate embodiment.

1. (canceled)
 2. A method comprising: receiving, from a mobile videodevice that is located in an environment, image data containing an imageof a marker at a location in the environment, wherein the marker has notbeen placed in the environment for electronic interaction with themobile video device; receiving a request from the mobile video devicefor information that locates an object with respect to the location ofthe marker; responsive to receiving the request, accessing a databasewith the image data to retrieve the requested information and retrievingthe requested information; generating a scaled image of the requestedinformation that locates the object, based on dimensions of an image ofthe environment and the location of the marker in the environment; andtransmitting the scaled image of the requested information that locatesthe object with respect to the location of the marker, for placementinto an image of the environment on a display of the mobile video deviceto generate an augmented reality image that comprises a path to theobject.
 3. The method of claim 2 wherein the scaled image of therequested information changes in accordance with locations in theenvironment that the mobile video device is focused on as the mobilevideo device moves in the environment and is focused on locations in theenvironment.
 4. The method of claims 2 and 3, wherein the objectcomprises a car.
 5. The method of claim 2 wherein the augmented realityimages comprises an image of a path to an exit from the environment thatis nearest to the location of the mobile video device.
 6. The method ofclaim 2 wherein the augmented reality image comprises an image of ashortest route from the location of the marker to the object.
 7. Themethod of claim 2 wherein the augmented reality image comprises an imageof a route from the environment to a first location, wherein the routeincludes a second location.
 8. The method of claim 2 further comprisingtransmitting advertisements to the mobile video device based onlocations in the environment that the mobile video device is focused onas the mobile video device moves in the environment and is focused onlocations in the environment.
 9. A method of augmenting a user's viewingexperience of a sporting event played by teams of players, the methodcomprising: receiving, from a user device located at an environment atwhich the user is viewing a sporting event, image data containing animage of the environment, and of a player on one of the teams playing inthe sporting event; receiving a request from the user device for an itemof information comprising statistics that relate to the player, forplacement into an image of the environment on the user device;responsive to receiving the request, performing recognition on the imageto recognize the player, and accessing a database to retrieve the itemof information for the player; generating a scaled image of the item ofinformation based on dimensions of the environment on the user device;and transmitting the scaled image to the user device for placement intothe image of the environment on the user device to generate an augmentedreality image.
 10. The method of claim 9 wherein request comprises animage of a uniform number of the player.
 11. The method of claim 9wherein the request comprises an image of the player's face and thestatistics relating to the player are determined using a facialrecognition algorithm.
 12. The method of claim 9 wherein the augmentedreality image comprises an arrow or other location indicator pointing tothe player.
 13. The method of claim 9 wherein the sporting event is abaseball game, and the item of information includes one of statistics ofrelevant teams, statistics by positions played by the players of therelevant teams, pitching rotation of pitchers playing for the relevantteams, current league standings of the relevant teams, and possibleplayoff scenarios for the relevant teams.
 14. The method of claim 9,further including targeting an advertisement to the user device based onthe user's purchase history, interest, or location.
 15. The method ofclaim 9 wherein playing the sporting event includes using a ball thatincludes an RFID chip, and RFID detection enables display of atrajectory of the ball.
 16. One or more computer-readable hardwarestorage device having stored therein a set of instructions which, whenexecuted by one or more processors of a computer, causes the computer toexecute operations comprising: receiving, from a user device located atan environment at which the user is viewing a sporting event, image datacontaining an image of the environment, and of a player in the sportingevent; receiving a request from the user device for an item ofinformation comprising statistics that relates to the player, forplacement into the image of the environment; responsive to receiving therequest, performing recognition on the image to recognize the player,and accessing a database to retrieve the item of information for theplayer; generating a scaled image of the item of information based ondimensions of the environment; and transmitting the scaled image to theuser device for placement into the image of the environment on the userdevice to generate an augmented reality image.
 17. The one or morecomputer-readable hardware storage device of claim 16 wherein therequest comprises an image of a uniform number of the player.
 18. Theone or more computer-readable hardware storage device of claim 16wherein the request comprises an image of the player's face and thestatistics relating to the player are determined using a facialrecognition algorithm.
 19. The one or more computer-readable hardwarestorage device of claim 16, the operations further comprising targetingan advertisement to the user device based on the user's purchasehistory, interest, or location.
 20. The one or more computer-readablehardware storage device of claim 16 wherein the augmented reality imagecomprises an arrow or other location indicator pointing to the player.21. The one or more computer-readable hardware storage device of claim16 wherein playing the sporting event includes using a ball thatincludes an RFID chip, and RFID detection enables display of atrajectory of the ball.